Analyzing the dynamic behavior and performance of a complex software system is difficult. Typically, analysis of a software system is achieved by gathering data at each system call and post-processing the data. Data is gathered at each system by placing a probe at locations of interest in the software (i.e., instrumenting the software to obtain an instrumented program) and gathering data when the probe is encountered by the thread executing the instrumented program.
Probes are typically represented in the instrumented code as trap instructions. The location (i.e., address) of each trap instruction is stored in a look-up table and associated with an original instruction (i.e., the instruction that was replaced when the program was instrumented to generate an instrumented program). The trap instruction is also typically associated with one or more actions that a tracing framework is to perform when the trap instruction is encountered by a thread executing the instrumented program.
When a thread executing the instrumented program encounters a trap instruction, control is transferred to a trap handler, which calls into the tracing framework and performs the action(s) associated with the trap instruction. The trap handler then looks up the original instruction in the look-up table. The trap instruction is then overwritten by the original instruction (i.e., the original instruction is placed in the same address space as the trap instruction in the code path). The tracing framework then single-steps the original instruction. The results of single-stepping the original instruction are then used to update the state of the instrumented program (i.e., the values of the registers, program counter(s), etc.) The original instruction in the code path is then overwritten by the trap instruction that was originally encountered by the thread. The thread then resumes executing the instrumented program.
Alternatively, the original instructions may be replaced with a reserved trap instruction, and when a thread executing the instrumented program encounters the reserved trap, all threads executing in the instrumented program are suspended while the thread that caused the trap single-steps the original instruction, which is temporarily written over the trap instruction, as defined above. After the thread has single-stepped the original instruction, the reserved trap that was encountered by the thread is copied back over the original instruction in the code path. All threads executing in the instrumented program then resume executing the instrumented program.
In a system in which more than one thread is executing within a given instrumented program, a particular thread may not trigger a probe (i.e., encounter a trap instruction) if the thread encounters the original instruction corresponding to a probe as opposed to the trap instruction. This situation typically occurs when a first thread encounters the trap instruction and overwrites it with a corresponding original instruction, and while this is occurring a second thread encounters the original instruction. In this scenario, the first thread calls into the tracing framework to perform the action(s) associated with the trap instruction, while the second thread executes the original instruction but does not call into the tracing framework.